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ABSTRACT 


Airspace  encounter  models  describe  encounter  situations  that  may  occur  between  aircraft  in  the 
airspace  and  are  a  critical  component  of  safety  assessment  of  sense  and  avoid  (SAA)  systems  for 
Unmanned  Aircraft  Systems  (UASs).  Some  UAS  will  fly  in  international  airspace  under  due  regard 
and  may  encounter  other  aircraft  during  these  operations.  In  these  types  of  encounters,  the  intruder 
aircraft  is  likely  receiving  air  traffic  control  (ATC)  services,  but  the  UAS  is  not.  Thus,  there  is  a 
need  for  a  due  regard  encounter  model  that  can  be  used  to  generate  these  types  of  encounters.  This 
report  describes  the  development  of  a  due  regard  encounter  model.  In  order  to  build  the  model, 
Lincoln  Laboratory  collected  data  for  aircraft  flying  in  international  airspace  using  the  Enhanced 
Traffic  Management  System  (ETMS)  data  feed  that  was  provided  by  the  Volpe  Center.  Lincoln 
processed  these  data,  and  extracted  important  features  to  construct  the  model.  The  model  is 
based  on  Bayesian  networks  that  represent  the  probabilistic  relationship  between  variables  that 
describe  how  aircraft  behave.  The  model  is  used  to  construct  random  aircraft  trajectories  that  are 
statistically  similar  to  those  observed  in  the  airspace.  A  large  collection  of  encounters  generated 
from  an  airspace  encounter  model  can  be  used  to  evaluate  the  performance  of  a  SAA  system  against 
encounter  situations  representative  of  those  expected  to  actually  occur  in  the  airspace. 

Lincoln  Laboratory  has  previously  developed  several  other  encounter  models.  There  is  an  uncor¬ 
related  encounter  model  that  is  used  to  generate  encounters  with  an  intruder  that  does  not  have  a 
transponder,  or  between  two  aircraft  using  a  Mode  A  code  of  1200  (VFR).  There  is  also  a  correlated 
encounter  model  that  is  used  when  both  aircraft  have  a  transponder  and  at  least  one  aircraft  is  in 
contact  with  ATC.  Both  of  these  models  were  built  from  radar  data  collected  from  the  National 
Airspace  System  (NAS).  There  is  also  an  unconventional  encounter  model  that  is  used  to  generate 
encounters  with  unconventional  intruders  such  as  gliders,  balloons,  and  airships — these  vehicles 
have  different  flight  characteristics  than  conventional  aircraft.  The  framework  used  to  construct 
the  due  regard  encounter  model  described  in  this  paper  is  similar  to  the  prior  models.  The  primary 
difference  is  that  a  different  data  feed  is  used  and  the  model  covers  encounters  in  international 
flight  where  the  aircraft  of  interest  is  flying  due  regard,  which  were  not  within  the  scope  of  prior 
models. 

Separate  electronic  files  are  available  from  Lincoln  Laboratory  that  contain  the  statistical  data 
required  to  generate  encounter  trajectories. 
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1.  INTRODUCTION 


Airborne  safety  critical  systems  must  undergo  extensive  validation  under  realistic  conditions  prior 
to  certification  in  the  National  Airspace  System  (NAS).  Collision  avoidance  systems  for  manned 
aircraft  and  sense  and  avoid  (SAA)  systems  for  Unmanned  Aircraft  Systems  (UASs)  provide  a 
safety  critical  function.  They  ensure  separation  when  other  safety  layers  have  failed  to  maintain 
separation  (e.g.,  the  underlying  airspace  structure,  airspace  procedures,  and  air  traffic  control). 
These  complex  systems  are  assessed  in  realistic  large  scale  Monte  Carlo  simulations  and  flight  tests 
to  prove  that  they  meet  the  desired  levels  of  safety.  Fundamental  to  these  simulations  is  the  use 
of  realistic  encounter  situations  between  aircraft  which  are  defined  by  the  relative  geometry  and 
behavior  of  the  aircraft  during  the  encounter.  The  geometry  and  dynamic  behavior  of  aircraft 
during  encounters  are  captured  in  encounter  models.  Encounter  models  provide  a  statistically 
sufficient  set  of  features  which  are  estimated  from  a  large  collection  of  observed  encounter  events. 
The  initial  and  continuing  evaluation  of  the  Traffic  Alert  and  Collision  Avoidance  System  (TCAS) 
for  manned  aircraft  illustrates  the  necessity  of  encounter  models  to  estimate  system  effectiveness 
in  a  wide  variety  of  encounter  geometries  [1-3] . 

Lincoln  Laboratory  has  previously  developed  encounter  models  for  the  NAS.  These  models  were 
built  from  radar  data  collected  from  radars  across  the  NAS  [4] .  There  are  fundamentally  two  types 
of  encounters:  correlated  and  uncorrelated.  In  the  first  type,  at  least  one  aircraft  in  the  encounter 
is  receiving  air  traffic  control  (ATC)  services,  and  both  aircraft  are  transponder  equipped.  In  the 
second  type,  ATC  services  would  no  longer  be  provided  because  either  one  aircraft  is  not  transpon¬ 
der  equipped  or  both  aircraft  are  flying  under  visual  flight  rules  (VFR).  The  first  type  is  termed 
“correlated”  because  there  is  active  coordination  provided  by  ATC  prior  to  the  loss  of  separation. 
It  is  therefore  critical  that  the  model  capture  this  coordination  between  aircraft  prior  to  the  loss  of 
separation.  In  the  second  type  of  encounter,  a  lack  of  ATC  services  results  in  uncoordinated  loss 
of  separation.  Hence,  these  encounters  are  termed  “uncorrelated” — i.e. ,  aircraft  blunder  into  one 
another.  This  uncorrelated  feature  is  exploited  by  modeling  each  aircraft  individually  and  then 
simulating  the  uncoordinated  loss  of  separation. 

This  report  describes  the  development  of  an  encounter  model  that  can  be  used  to  evaluate  SAA 
systems  operating  on  UAS  flying  due  regard  in  oceanic  airspace.  An  aircraft  flying  due  regard 
is  not  operating  under  International  Civil  Aviation  Organization  (ICAO)  flight  procedures  and  is 
responsible  for  maintaining  its  own  separation  from  other  aircraft  [5] .  More  specifically,  a  due  regard 
encounter  model  describes  the  statistical  distribution  of  close  encounter  situations  that  are  expected 
to  occur  in  oceanic  airspace  when  an  UAS  is  operating  due  regard.  A  due  regard  encounter  model 
is  an  uncorrelated  model  because  ATC  intervention  is  unlikely.  The  next  section  gives  an  overview 
of  all  the  encounter  models  that  Lincoln  has  developed  and  describes  when  it  is  appropriate  to  use 
each  one.  The  following  section  then  gives  an  overview  of  this  report. 


1.1  MODEL  SELECTION 

UASs  are  envisioned  to  operate  in  a  variety  of  environments.  As  an  example,  the  Broad  Area 
Maritime  Surveillance  (BAMS)  UAS  will  operate  over  the  contiguous  United  States,  in  offshore 
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environments,  and  in  oceanic  environments  [6] .  Lincoln  has  developed  a  variety  of  encounter  models 
to  evaluate  SAA  systems  as  a  result  of  the  wide  range  of  operating  environments  for  UASs.  There 
are  four  types  of  encounter  models: 


•  Uncorrelated  Encounter  Model  of  the  National  Airspace  System  (U):  An  uncor¬ 
related  encounter  model  is  used  to  evaluate  the  performance  of  SAA  systems  when  at  least 
one  aircraft  is  noncooperative  or  neither  aircraft  are  in  contact  with  ATC  [7].  This  model 
was  developed  from  1200-code  aircraft  observed  in  the  NAS  and  this  model  now  includes  the 
offshore  environment  out  to  the  limits  of  radar  coverage  [8] . 

•  Correlated  Encounter  Model  of  the  National  Airspace  System  (C):  A  correlated 
encounter  model  is  used  to  evaluate  the  performance  of  SAA  systems  when  both  aircraft  are 
cooperative  and  at  least  one  aircraft  is  receiving  ATC  services  [9] .  This  model  was  developed 
from  observed  encounters  in  the  NAS  and  also  includes  the  offshore  environment. 

•  Encounter  Models  for  Unconventional  Aircraft  (X):  This  encounter  model  is  used  to 
evaluate  the  performance  of  an  SAA  system  when  encountering  unconventional  aircraft,  de¬ 
fined  as  aircraft  unlikely  to  carry  a  transponder  [10].  Examples  of  unconventonial  aircraft  in¬ 
clude  balloons,  blimps,  gliders,  and  skydivers.  This  model  was  developed  from  GPS-recorded 
tracks  that  are  posted  online. 

•  Due  Regard  Encounter  Model  (D):  This  encounter  model  is  used  to  evaluate  SAA 
systems  operating  on  UAS  flying  due  regard  in  oceanic  airspace.  This  model  is  primarily 
built  from  self-reported  positions  of  aircraft  flying  in  oceanic  airspace. 


Each  of  these  encounter  models  includes  variables  that  account  for  variations  in  encounters  with 
respect  to  different  airspaces.  For  example,  one  of  the  variables  in  the  uncorrelated  encounter  model 
is  Airspace  Class,  which  includes  B,  C,  D,  and  O  (Other).  Table  1  indicates  the  appropriate  model 
to  use  based  on  the  study  being  performed.1  For  the  offshore  environment,  the  correlated  and 
uncorrelated  encounter  models  encompass  encounters  more  than  1 NM  beyond  the  shore  and  the 
due  regard  model  begins  at  12  NM,  where  due  regard  flight  is  permitted.2  The  oceanic  environment 
includes  international  airspace  beyond  radar  coverage.  Note  that  no  existing  model  covers  encoun¬ 
ters  between  two  IFR  aircraft  in  oceanic  airspace.  The  reason  for  this  is  that  one  cannot  observe 
encounters  of  sufficient  fidelity  in  the  data  feeds.  Similarly,  there  is  no  model  that  covers  encounters 
with  VFR  or  noncooperative  aircraft  in  oceanic  airspace  due  to  a  lack  of  surveillance  data.  If  a 
collection  of  encounters  for  these  types  of  encounters  is  required,  they  should  be  built  based  on  best 
assumptions  about  aircraft  behavior  and  should  leverage  data  from  similar  encounter  models  as  is 
necessary.  For  example,  one  could  use  offshore  models  or  enroute  model  of  the  Continental  United 
States  (CONUS). 

xNote  that  a  model  does  not  exist  for  combinations  without  a  mark. 

2Note  that  one  cannot  build  a  correlated  encounter  model  from  radar  data  for  due  regard  flight  in  the  offshore 
environment  because  one  does  not  observe  a  sufficient  number  of  encounters  between  instrument  flight  rules  (IFR) 
and  non-IFR  traffic  beyond  12  NM. 


2 


TABLE  1 


Encounter  model  categories. 


Aircraft  of  Interest 

Intruder  Aircraft 

Location 

Flight  Rule 

IFR 

VFR 

Noncooperative 

Conventional 

Noncooperative 

Unconventional 

IFR 

C 

C 

U 

X 

CONUS 

VFR 

C 

u 

U 

X 

IFR 

c 

c 

u 

X 

Offshore 

VFR 

c 

u 

u 

X 

Due  Regard 

0 

u 

u 

X 

Oceanic 

IFR 

VFR 

Due  Regard 

1.2  OVERVIEW 

This  document  describes  the  due  regard  encounter  model.  Section  2  describes  the  Enhanced  Traffic 
Management  System  (ETMS)  data  used  to  build  the  model.  Section  3  describes  the  model  itself. 
Section  4  describes  how  to  process  the  ETMS  data  to  build  the  due  regard  encounter  model. 
Section  5  discusses  some  aspects  of  safety  evaluation  using  the  encounter  model,  which  includes  the 
process  for  initializing  encounters  and  calculating  the  probability  of  near  mid-air  collision  (NMAC) 
given  an  encounter.  Lastly,  Section  6  summarizes  the  model. 

Note  that  the  overall  development  process  for  this  model  closely  follows  that  for  the  uncorrelated 
encounter  model  for  the  NAS.  As  is  such,  some  development  details  and  background  are  omitted 
from  this  report.  The  reader  should  refer  to  the  uncorrelated  model  technical  report  for  more 
information  [7]. 
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2.  DATA 


ETMS  is  used  by  the  Federal  Aviation  Administration  (FAA)  to  support  the  strategic  flow  of  aircraft 
operating  in  terrestrial  and  oceanic  airspace.  ETMS  processes  data  from  a  variety  of  sources  that 
include  oceanic  position  updates,  NAS  messages,  weather  data,  airline  schedules,  traffic  data,  and 
others.  This  information  is  then  disseminated  to  support  various  traffic  management  functions 
such  as  traffic  display,  congestion  predictions,  congestion  management,  rerouting,  and  collaborative 
decision  making  [11]. 

Lincoln  Laboratory  maintains  an  ETMS  data  feed  (now  the  Traffic  Flow  Management  System)  and 
uses  a  subset  of  the  track  data  over  oceanic  airspace  available  in  the  feed  to  build  the  due  regard 
encounter  model.  Track  data  is  updated  once  a  minute  in  ETMS  and  includes  reported  position, 
altitude,  ground  speed,  and  heading.  Construction  of  the  model  used  eight  weeks  during  2009 
which  were  distributed  throughout  the  year  to  prevent  model  bias  due  to  seasonal  variations.  The 
collection  periods  were: 

•  1  February  2009  through  14  February  2009, 

•  3  May  2009  through  16  May  2009, 

•  2  August  2009  through  14  August  2009,  and 

•  9  November  2009  through  22  November  2009. 

After  extracting  tracks  in  the  oceanic  environment,  10,579  hours  of  track  data  were  used  to  build 
the  due  regard  encounter  model.  Figure  1  shows  the  average  number  of  aircraft  observed  in  the 
ETMS  data  within  half  degree  by  half  degree  bins  across  international  airspace  during  the  collection 
period.  Only  data  for  cells  within  one  of  the  geographic  domains  for  the  due  regard  encounter  model 
are  presented.  Cells  with  no  data  are  white. 

Previous  encounter  models  have  been  built  from  radar  data  with  five  or  twelves  update  rates.  Due 
to  the  fact  that  ETMS  tracks  are  updated  once  every  60s,  it  was  necessary  to  first  validate  that 
ETMS  data  can  be  used  to  build  an  encounter  model.  This  validation  is  described  in  Appendix  B. 
The  high-level  approach  involves  building  a  site-specific  encounter  model  from  both  radar  and 
ETMS  data  from  the  same  geographic  region  that  includes  many  aircraft  tracks  in  the  North 
Atlantic  Tracks  (NAT).  The  models  are  built  and  compared  to  show  that  they  are  statistically 
similar. 
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Figure  1.  Average  number  of  aircraft  present  in  half  degree  by  half  degree  bins  during  data  collection  period. 
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3.  MODEL 


Nominal  flight,  i.e.,  flight  without  avoidance  maneuvering,  is  modeled  using  a  Markov  process 
represented  by  a  dynamic  Bayesian  network.  A  Markov  process  is  a  stochastic  process  where  the 
probability  distribution  over  future  states  is  conditionally  independent  of  past  states  given  the 
present  state.  In  other  words,  one  only  needs  to  know  the  present  state  to  predict  the  next  state. 

The  states  in  the  model  specify  how  the  position,  altitude,  and  airspeed  change  over  time.  In 
particular,  each  state  specifies  a  vertical  rate  h,  turn  rate  iJj,  and  airspeed  acceleration  v.  Given  an 
initial  airspeed  v,  position,  heading  i/;,  vertical  rate  h,  altitude  layer  L,  and  geographic  domain  G, 
one  can  infer  from  the  model  how  the  aircraft  trajectory  evolves  over  time. 

One  way  to  represent  a  Markov  model  is  with  an  exhaustive  state-transition  matrix  that  specifies  the 
probability  of  transitioning  between  all  pairs  of  states.  Unfortunately,  the  number  of  independent 
parameters  required  to  define  the  matrix  grows  super  exponentially  with  the  number  of  variables 
defining  the  model.  The  more  independent  parameters  that  are  in  the  model,  the  more  data 
one  needs  to  properly  estimate  their  values.  However,  by  using  dynamic  Bayesian  networks,  one 
can  leverage  conditional  independence  between  some  variables  to  greatly  reduce  the  number  of 
parameters.  One  can  learn  the  structure  of  the  dynamic  Bayesian  network  by  maximizing  the 
posterior  probability  of  the  network  structure  given  the  data  [4] . 

The  Bayesian  networks  that  maximized  the  posterior  probability  of  the  structure  can  be  determined 
by  searching  the  space  of  network  structures.  The  general  process  involves  guessing  a  fully  connected 
network  structure  and  scoring  that  structure.  A  collection  of  networks  is  then  constructed  where  for 
each  network  a  single  conditional  dependency  is  removed  between  two  variables  in  the  network  and 
score  them.  If  any  of  the  new  networks  has  a  higher  probability,  it  is  kept  and  the  search  continues. 
The  search  terminates  once  removing  any  conditional  dependency  in  the  network  decreases  the 
probability. 

3.1  MODEL  VARIABLES 

There  are  seven  variables  in  the  due  regard  encounter  model: 

•  Altitude  layer  L:  Airspace  is  divided  into  six  altitude  layers,  in  a  process  similar  to  prior 
encounter  models.  The  first  layer  spans  up  to  5500  ft  Mean  Sea  Level  (MSL).  The  second 
layer  spans  5500  to  10,000  ft  MSL.  The  third  layer  spans  from  10,000  ft  MSL  to  FL180.  The 
fourth  through  sixth  layers  are  FL180  to  FL290,  FL290  to  FL410,  and  above  FL410. 

•  Geographic  Domain  G:  The  due  regard  encounter  model  is  separated  into  domains  to 
capture  variations  in  aircraft  behavior  as  a  result  of  geographic  area.  This  is  a  new  variable 
for  encounter  modeling,  which  is  discussed  in  more  detail  below. 

•  Airspeed  v:  True  airspeed  is  modeled  and  allowed  to  vary  during  flight.  Units  are  in  kts. 

•  Heading  -0:  Aircraft  heading  is  modeled  in  the  due  regard  encounter  model  and  it  was 
allowed  to  vary  during  flight.  This  variable  is  unique  to  the  due  regard  encounter  model 
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because  of  the  unique,  regular  route  structures  in  much  of  international  airspace.  Units  are 
in  degrees 

•  Acceleration  v:  Airspeed  acceleration  is  allowed  to  vary  every  second.  Units  are  in  kts/s. 

•  Turn  rate  ip:  Turn  rate  is  permitted  to  change  every  second  in  the  model.  Units  are 
degrees/s. 

•  Vertical  rate  h:  The  vertical  rate  is  permitted  to  change  at  every  second.  Units  are  ft/rnin. 

Because  many  of  the  variables  are  closely  related  due  to  physical  constraints  and  flight  character¬ 
istics  (e.g.,  turn  rate  and  vertical  rate),  it  is  important  to  properly  represent  correlations  in  the 
model.  Independently  sampling  from  distributions  for  turn  rate  and  vertical  rate  would  ignore 
these  important  relationships.  The  remainder  of  this  section  explains  how  to  model  joint  probabil¬ 
ity  distributions  over  these  variables  to  ensure  proper  consideration  of  correlations.  To  generate  an 
encounter,  one  first  randomly  samples  from  the  joint  distribution  over  the  encounter  variables  to 
define  the  initial  conditions.  A  Markov  model  is  then  used  to  determine  how  the  dynamic  variables 
(i.e. ,  turn  rate,  vertical  rate,  and  airspeed  acceleration)  evolve  during  the  course  of  the  encounter. 
There  are  two  corresponding  separate  probability  distributions  in  the  model:  an  initial  distribution 
to  set  up  an  encounter  situation  and  a  transition  distribution  to  describe  how  the  dynamic  variables 
specifying  the  trajectories  of  the  aircraft  evolve  over  time. 

The  geographic  domains  in  the  due  regard  encounter  model  are  listed  in  Table  2  and  are  depicted 
in  Figure  2.  Table  2  also  shows  the  longitudinal  and  latitudinal  limits  for  the  geographic  domains 
in  the  due  regard  encounter  model.  Note  that  the  limits  only  apply  in  international  airspace — 
e.g.,  airspace  greater  than  12  NM  from  land.  Thus,  aircraft  in  one  of  these  domains  but  over  land 
are  not  part  of  the  model.  The  upper  longitudinal  limit  (66°33/44,/  N)  corresponds  to  the  arctic 
circle.  These  domains  were  selected  because  they  encompass  U.S. -controlled  oceanic  airspace  and 
they  broadly  capture  areas  with  repeatable  and  similar  tracks — e.g.,  tracks  with  similar  magnetic 
headings. 


TABLE  2 


Geographic  domain  limits. 


Longitudinal  Limits 

Latitudinal  Limits 

1 

North  Pacific 

37°0/0"  N  to  66°33/44"  N 

140°0'0"  E  to  98°0'0"  W 

2 

West  Pacific 

15°0'0"  S  to  37°0/0"  N 

140°0/0//  E  to  158°0/0//  W 

3 

East  Pacific 

15°0'0"  S  to  37°0/0"  N 

158°0/0//  W  to  99°0/0//  W 

4 

Gulf  of  Mexico 

16°0'0"  N  to  30°48'  N 

98°0'0"  W  to  82°0/0"  W 

5 

Caribbean 

7°30/0"  N  to  37°0/0//  N 

82°0/0"  W  to  60°0/0//  W 

6 

North  Atlantic 

37°0/0"  N  to  66°33/44"  N 

82°0,0"  W  to  0°0'0" 

7 

Central  Atlantic 

15o0'0"  S  to  37°0/0"  N 

60°0'0"  W  to  15°0/0//  E 

Figure  2.  Geographic  domains. 


3.2  INITIAL  DISTRIBUTION 

It  is  important  that  the  model  of  aircraft  flight  captures  the  distribution  over  the  initial  values  of 
h,  ip,  v,  ip,  v,  L,  and  G.  Figure  3  shows  the  structure  used  for  the  modeling  effort.  Other  structures 
are  certainly  plausible,  and  it  is  possible  to  compare  different  structures  using  Bayesian  scoring  to 
determine  which  structure  is  more  likely  given  the  data  [4],  The  network  structure  in  Figure  3  was 
chosen  because  it  was  the  most  likely  among  the  selection  of  candidate  networks  considered  in  the 
structure  search.  Appendix  A  contains  figures  showing  marginal  distributions  for  the  individual 
variables  in  the  model. 

Given  a  structure,  sufficient  statistics  extracted  from  data,  and  a  Bayesian  prior,  one  can  sample 
from  the  Bayesian  network  to  produce  sets  of  initial  airspace  classes,  altitude  layers,  vertical  rates, 
turn  rates,  airspeeds,  and  accelerations  that  are  representative  of  those  found  in  the  data.  The 
boxes  and  arrows  used  in  the  structural  diagram  show  the  order  in  which  this  sampling  occurs. 
For  example,  based  on  the  structure  in  Figure  3,  to  determine  the  initial  state  of  the  aircraft  one 
would  first  randomly  determine  an  altitude  layer  L.  Once  the  altitude  layer  has  been  determined,  a 
geographic  domain  G  is  selected.  The  probabilities  associated  with  each  geographic  domain  depend 
on  which  altitude  layer  was  chosen  earlier.  Once  L  and  G  have  been  selected,  the  model  randomly 
selects  airspeed  v,  and  so  on.  Alternately,  one  could  assign  outright  a  geographic  domain  and/or 
altitude  layer  for  a  particular  study  and  then  randomly  select  values  for  the  remaining  variables. 


3.3  TRANSITION  DISTRIBUTION 

A  separate  Bayesian  network  is  used  to  model  how  the  variables  h,  ip,  and  v  evolve  over  time 
(see  Figure  4).  In  this  network,  the  first  layer  represents  the  state  of  the  trajectory  at  the  present 
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Figure  3.  Initial  structure. 


time  step  and  a  second  layer  that  represents  the  state  of  the  trajectory  at  the  next  time  step. 
There  may  be  dependencies  between  layers  and  within  the  second  layer.  Such  a  two-layer  temporal 
Bayesian  network  is  known  as  a  dynamic  Bayesian  network  [12].  Parameter  and  structure  learning 
in  dynamic  Bayesian  networks  is  similar  to  regular  Bayesian  networks.  Again,  the  highest-scoring 
network  structure  was  chosen  among  the  candidate  network  structures.  Given  a  structure,  sufficient 
statistics  extracted  from  data,  and  a  prior,  one  then  samples  from  the  Bayesian  network  to  determine 
the  next  vertical  rate,  turn  rate,  and  acceleration  command  that  are  representative  of  what  was 
observed  in  the  data. 

In  general,  time  steps  in  dynamic  Bayesian  networks  may  be  of  any  duration,  but  for  the  encounter 
modeling  effort  a  time  step  of  1  s  was  used.  Shorter  time  steps  allow  for  more  frequent  variations 
in  airspeed,  vertical  rate,  and  turn  rate,  but  they  require  more  computation  per  unit  of  simulation 
time.  Time  steps  of  1  s  balance  maneuver  complexity  with  computation. 
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Figure  4-  Transition  structure. 


A  complete  trajectory  is  constructed  by  updating  the  aircraft  state  in  1  s  intervals.  Within  each 
interval,  the  three  derivative  variables  h,  iJj,  and  i)  are  treated  as  target  values  and  held  constant. 
A  dynamic  model  (which  is  beyond  the  scope  of  this  report)  is  used  to  compute  and  update  the 
aircraft  state  at  each  time  step  based  on  these  piecewise-constant  target  values.  The  encounter 
model  can  generate  statistically  representative  trajectories  that  persist  for  more  than  1000  s.  In 
order  to  determine  this,  a  statistical  test  of  the  encounter  model  was  performed,  which  is  described 
in  Appendix  C. 
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4.  PROCESSING 


Figure  5  shows  the  processing  flow  to  build  the  due  regard  encounter  model.  The  first  step  was 
spatially  filtering  tracks  to  extract  tracks  in  international  airspace.  The  next  step  involves  removing 
outliers.  Next,  the  tracks  are  interpolated  and  the  model  features  are  extracted.  The  features  are 
then  quantized  and  extracted  to  estimate  the  sufficient  statistics.  The  remainder  of  this  section 
describes  this  process  in  more  detail. 


Figure  5.  Estimation  process  flow. 
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4.1  OUTLIER  REMOVAL 


There  are  some  outliers  in  the  ETMS  data.  Thus,  these  outliers  first  have  to  be  detected  and 
removed  so  that  the  model  accurately  reflects  aircraft  motion.  Outliers  are  detected  in  altitude, 
ground  speed,  and  heading: 

•  Altitude:  A  reported  altitude  is  assumed  to  be  an  outlier  if  the  change  in  altitude  is  more 
than  10,000  ft  over  one  minute.  Some  altitudes  are  reported  as  0  ft — these  are  assumed  to  be 
outliers,  as  well. 

•  Ground  Speed:  A  reported  ground  speed  is  assumed  to  be  an  outlier  if  the  speed  is  less 
than  50  kts  or  greater  than  650  kts.  If  the  change  in  ground  speed  is  greater  than  120  kts 
between  60  s  reports,  then  it  is  assumed  to  be  an  outlier,  as  well. 

•  Heading:  A  reported  heading  is  assumed  to  be  an  outlier  if  a  heading  is  reported  as  0  or 
360  degrees  and  the  change  in  heading  is  more  than  60  degrees  between  the  previous  or  next 
report.  For  example,  if  a  sequence  of  reported  headings  were  88,  88, 87, 0,  87,  87,  then  the  0  is 
removed. 

Processing  also  compares  the  reported  ground  speed  and  heading  against  values  that  are  calculated 
from  consecutive  position  reports.  If  the  calculated  ground  speed  difference  is  greater  than  100  kts, 
then  an  outlier  is  assumed.  If  the  calculated  heading  difference  is  greater  than  25  degrees,  then  an 
outlier  is  also  assumed. 


4.2  INTERPOLATION 

Next,  the  altitude,  ground  speed,  and  headings  are  interpolated.  Interpolation  is  done  at  1  s  using 
a  shape  preserving  piecewise-cubic  interpolation.  Note  that  interpolation  does  not  occur  between 
reports  if  there  is  a  missing  report  between  them-  i.e. ,  the  time  between  reports  is  greater  than 
60  s. 

4.3  FEATURE  EXTRACTION 

Recall  that  the  variables  in  the  model  are  L,  G,  v,  ip,  v,  ip,  and  h.  The  variable  L  is  extracted 
directly  from  the  altitude  reports.  The  variable  G  is  estimated  using  the  nearest  reported  latitude 
and  longitude.  Airspeed  is  assumed  to  be  equivalent  to  ground  speed.  The  variable  ip  is  extracted 
directly  from  heading.  Finally,  v,  ip,  and  h  are  extracted  by  calculating  the  derivative  of  ground 
speed,  heading,  and  altitude,  respectively. 

4.4  QUANTIZATION 

In  order  to  be  modeled  by  a  discrete  Bayesian  network,  it  is  necessary  to  quantize  the  features. 
Continuous  values  are  quantized  by  defining  a  sequence  of  cut  points  ci, . . .  ,cn.  Values  less  than 
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ci  are  in  the  first  bin,  values  greater  than  cn  are  in  the  (n  +  l)th  bin,  and  values  in  the  half-open 
interval  [cj_ i,  Cj)  are  in  the  ith  bin.  The  cut  points  used  for  quantization  are  listed  in  Table  3. 


TABLE  3 

Cut  points  used  for  feature  quantization. 


Cut  Points 

L  5500  ft,  10000  ft,  FL180,  FL290,  FL410 
G  1,  2,  3,  4,  5,  6,  7 
v  200,  300,  400,  500 
60,  120,  180,  240,  300 
v  -1.5,  -0.25,  0.25,  1.5 

h  -3000,  -2000,  -1000,  -400,  400  1000,  2000,  3000 
■0  -3.5,  -1,  -0.25,  0.25,  1,  3.5 


4.5  STATISTICS  EXTRACTION 

With  structures  for  the  initial  and  transition  distributions  and  the  quantized  features  from  a  set  of 
tracks,  the  sufficient  statistics  can  be  collected  to  estimate  the  parameters  for  the  model.  For  the 
two  Bayesian  networks,  the  sufficient  statistics  are  simply  the  counts  of  the  various  features  (see 
Appendix  I  in  [7]). 3  Appendix  I  in  [7]  describes  the  sufficient  statistics  and  Bayesian  networks  in 
greater  detail. 


3The  counts  are  called  sufficient  statistics  because  they  provide  a  summarization  of  the  data  that  is  sufficient  to 
compute  the  posterior  distribution  from  the  prior.  For  an  introduction  to  Bayesian  statistics,  see  [13]. 
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5.  SAMPLING,  SIMULATION,  AND  SAFETY  EVALUATION 


Once  the  data  have  been  processed  as  described  in  the  previous  section,  one  can  use  the  model  to 
produce  new  trajectories  that  are  representative  of  those  observed  in  the  ETMS  data  by  sampling 
from  the  Bayesian  networks.  This  process  is  carefully  described  in  Section  4  of  [7].  The  next  step 
is  to  initialize  the  trajectory  of  an  intruder  aircraft  on  the  encounter  cylinder  centered  about  the 
aircraft  of  interest.  Section  5  of  [7]  describes  this  process.  However,  there  are  some  important 
changes  in  this  model.  Thus,  Section  5.1  of  this  document  describes  the  process  to  initialize 
encounter  situations  that  should  be  used  with  this  model.  Section  6  of  [7]  then  described  how  to 
estimate  NMAC  probability.  Due  to  the  fact  that  a  different  initialization  process  is  used  in  this 
model,  the  appropriate  procedure  for  estimating  NMAC  probability  is  different.  It  is  discussed  in 
Section  5.2  of  this  document. 


5.1  ENCOUNTER  INITIALIZATION 

Rejection  sampling  is  used  to  generate  the  initial  conditions  of  an  encounter.  Rejection  sampling 
involves  proposing  a  series  of  candidate  samples  from  a  random  distribution  until  choosing  one 
that  meets  a  set  of  criteria.  The  process  used  for  generating  initial  conditions  for  encounters  is  as 
follows: 

1.  Generate  a  set  of  initial  conditions  for  the  due  regard  aircraft  (AC1).  The  initial  conditions 
include  altitude  layer,  geographic  domain,  airspeed,  heading,  vertical  rate,  turn  rate,  and 
acceleration. 

2.  Sample  from  the  Bayesian  network  to  generate  AC2.  If  the  sample  has  the  same  altitude  layer 
and  geographic  domain,  then  keep  AC2;  otherwise,  reject  the  sample  and  continue  to  sample 
until  the  altitude  layer  and  geographic  domain  match  AC1.4  AC1  and  AC2  are  termed  the 
own  and  intruder  aircraft,  respectively. 

3.  Calculate  the  velocity  vectors  for  both  aircraft  (vi  and  V2)  based  on  the  aircraft  initial 
conditions.  The  relative  velocity  vector  vr  is  vi  subtracted  from  V2.  See  Figure  6. 

4.  Determine  the  projected  surface  S  of  the  cylinder  onto  a  plane  that  is  perpendicular  to  vr. 

5.  The  intruder  aircraft  penetrates  the  encounter  cylinder  uniformly  over  S.  Therefore,  uni¬ 
formly  select  a  random  point  p  inside  S. 

6.  Project  this  point  back  onto  the  encounter  cylinder.  There  will  be  two  candidate  points. 
Select  the  one  such  that  the  intruder  aircraft  is  penetrating  the  encounter  cylinder.  The 
following  tests  can  be  used  to  determine  which  candidate  initial  point  is  correct: 

•  If  AC2  was  initialized  on  the  top  of  the  encounter  cylinder,  accept  the  sample  if  the 
vertical  rate  of  AC2  relative  to  AC1,  denoted  vrjV,  is  negative.  This  ensures  that  AC2 
is  penetrating  the  encounter  cylinder  for  the  first  time. 

4Note  that  this  will  result  in  an  incorrect  distribution  over  layer  and  geographic  domain  (compared  to  that 
observed).  Section  5.3  describes  how  to  correct  for  this. 
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•  If  AC2  was  initialized  on  the  bottom  of  the  encounter  cylinder,  accept  the  sample  if  the 
vertical  rate  of  AC2  relative  to  AC1,  denoted  vrjV,  is  positive.  This  ensures  that  AC2  is 
penetrating  the  encounter  cylinder  for  the  first  time. 

•  If  AC2  was  initialized  on  the  side  of  the  encounter  cylinder,  accept  the  sample  if  Rh  •  vr-h 
is  negative.  Here  Rh  is  the  horizontal  component  of  R,  which  is  the  relative  position  of 
AC2  from  AC1,  and  vr  h  is  vi  h  subtracted  from  V2,h-  The  vectors  Vi  h  and  V2,h  are  the 
horizontal  velocities  of  AC1  and  AC2  respectively.  If  Rh  •  vr.h  is  negative,  then  accept 
the  encounter  because  AC2  is  penetrating  the  cylinder  about  AC1. 


Figure  6.  Initialization  process. 


5.2  ESTIMATING  NMAC  PROBABILITY 

Section  5.1  explained  how  to  construct  an  encounter  from  two  independent  trajectories  sampled  from 
the  distribution  represented  by  Bayesian  networks.  By  generating  a  large  collection  of  encounters 
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and  determining  which  encounters  lead  to  NMACs,  one  can  estimate  P(nmae  |  enc).  Unfortunately, 
one  cannot  simply  divide  the  number  of  sampled  encounters  that  lead  to  NMACs  by  the  total 
number  of  sampled  encounters  to  estimate  P(nmac  |  enc)  due  to  the  fact  that  the  sampling  scheme 
does  not  produce  encounters  from  the  same  distribution  that  would  occur  in  the  airspace.  In 
particular,  the  model  generates  encounters  with  aircraft  velocities  distributed  identically  to  the 
aircraft  population  at  large,  despite  the  fact  that  in  reality  the  distribution  of  aircraft  velocities 
given  that  an  encounter  is  occurring  favors  high-speed  aircraft.  Although  one  samples  from  a 
distribution  that  is  different  from  the  true  distribution  when  constructing  encounters,  one  can  still 
use  the  samples  to  estimate  P(nmac  |  enc)  so  long  as  one  weights  their  results  properly  using  an 
approach  known  as  importance  sampling  [14] .  This  section  begins  by  stating  the  weighting  scheme 
and  then  proves  that  it  is  correct. 


Section  4  of  [7]  explains  how  to  generate  the  trajectories  for  AC1  and  AC2,  z\  and  z2,  by  sampling 
from  the  Bayesian  networks  with  the  requirement  that  both  aircraft  come  from  the  same  altitude 
layer  and  geographic  domain.  Section  5.1  then  explained  how  to  randomly  select  the  position  and 
orientation  of  AC2  relative  to  AC1,  which  is  termed  xr.  Importance  sampling  allows  one  to  make 
the  following  approximation  based  on  N  samples 


P(nmac  |  enc) 


—  V  Pfnrnac 
N  4^  v 


,(*)  *(*)  xw 


■‘l  i  2  ) 
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The  weight  V(z^\z2^)/V  corrects  for  the  fact  that  the  sampling  distribution  does  not  match 

the  true  distribution  of  encounter  situations.  The  function  U(z^,z®)  is  the  average  volume 

(i) 

the  encounter  cylinder  sweeps  out  per  unit  time  when  AC1  follows  z\  and  the  airspace  consists 

(i) 

exclusively  of  aircraft  following  z\  .  In  particular, 

V(z i  \z^)  =  |vr|  a(S) , 


where  |ur|  is  the  magnitude  of  the  relative  velocity  vectors  and  a(S)  is  the  area  of  S.  The  constant 
V  is  the  average  volume  the  encounter  cylinder  sweeps  out  per  unit  time 


V=  p(z1)p(z2  |  zi)U(zi,  z2)  dzidz2  . 


Note  that  the  distribution  over  z2  is  conditional  on  zi  due  to  the  constraint  that  AC1  and  AC2 
must  belong  to  the  same  airspace  class  and  altitude  layer.  The  constant  V  can  be  estimated  using 
N  samples: 


V 


N 


(i) 
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Now  that  the  weighting  scheme  has  been  defined,  this  section  now  proves  that  it  is  correct.  From 
the  laws  of  probability, 


P(nmac  |  enc)  =  //  P(nmac  |  zi, z2, enc)p(zi, z2  |  enc)  dzidz2 

P(nmac  |  zi,  z2,  xr,  enc)p(x.r  |  z\,  z2,  enc)p(zi,  z2  |  enc)  dzidz2dxr 
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P(nnrac  |  enc)  may  be  approximated  using  Monte  Carlo  sampling.  Since  it  is  difficult  to  sample 
from  p( zi,Z2  |  enc)  directly,  one  can  sample  z\  and  z2  from  the  distribution  represented  by  the 
Bayesian  network  subject  to  the  constraint  that  both  aircraft  come  from  the  same  geographic 
domain  and  altitude  layer,  and  weight  the  samples  appropriately: 

enc) 
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This  result  may  be  normalized  to  obtain 
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Substitution  and  simplification  leads  to 
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which  corresponds  to  the  weighting  scheme  defined  above. 


5.3  CORRECTING  FOR  LAYER  AND  GEOGRAPHIC  DOMAIN 

Encounters  are  generated  across  altitude  layers  and  geographic  domains  according  to  their  dis¬ 
tributions  defined  in  the  model.  These  distributions  represent  the  observed  rate  of  occurrence  of 
aircraft  in  each  altitude  layer  and  geographic  domain.  However,  the  expected  encounter  rate  is  pro¬ 
portional  to  the  airspace  density,  not  the  cumulative  occurrence  of  aircraft,  in  the  local  airspace. 
Furthermore,  the  aircraft  of  interest  may  be  exposed  to  distribution  of  encounters  in  altitude  layers 
and  geographic  domains  that  is  different  than  the  distributions  in  the  model  because  the  aircraft 
of  interest  has  a  greater  exposure  time  (te)  to  encounters  in  other  altitude  layer  and  geographic 
domain  combinations.  Therefore,  the  mean  NMAC  probability  over  all  geographic  domains  and 
altitude  layers  that  considers  these  factors  a  posteriori  is 

J2i  P(nmac  |  enc,  at,  li 

Ei  PiVite^ 


P(nnrac  |  enc)  =  ^  P(nmac  |  enc ,  ctj,  4)P(aj,  Z*  |  enc) 
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where  i  denotes  each  altitude  layer  and  geographic  domain  combination.  The  term  P(li,gi  \  enc)  is 
the  proportion  of  encounters  expected  in  each  altitude  layer  and  geographic  domain  combination. 

If  one  knows  pt,  Vi,  and  te  '  a  priori,  then  the  altitude  and  airspace  class  distributions  can  be  modified 
to  reflect  this  knowledge  before  sampling  from  the  model.  Then,  the  mean  NMAC  probability 
estimate  is  simply  the  NMAC  probability  for  all  encounters.  This  sampling  procedure  may  be 
useful  when  the  aircraft  of  interest  is  expected  to  operate  in  a  specific  geographic  domain  or  altitude 
layer.  If  little  is  known  about  the  expected  operating  environment,  then  an  objective  (i.e. ,  uniform) 
assumption  regarding  the  geographic  domain  and  altitude  layer  may  be  suitable. 


5.4  ALTERNATIVE  DISTRIBUTIONS 

Some  analyses  may  require  that  encounters  be  generated  from  an  alternative  distribution  than  that 
produced  by  the  model  using  the  procedure  described  earlier  in  this  section.  For  example,  one  may 
expect  a  specific  distribution  over  headings  or  horizontal  miss  distance  (HMD)  that  is  different 
than  that  generated  by  the  model.  This  may  be  a  result  of  a  specific  operation,  for  example.  This 
section  describes  how  to  account  for  alternative  distributions. 

There  are  two  relevant  approaches.  The  first  involves  employing  a  procedure  for  manipulating  the 
Bayesian  network,  which  is  only  applicable  in  some  circumstances.  Another  approach  called  likeli¬ 
hood  weighting  [15]  involves  weighting  the  samples  to  correct  for  the  fact  the  sampled  distribution 
does  not  match  the  expected  distribution. 

5.4.1  Bayesian  Network  Manipulation 

If  one  samples  a  subset  of  the  encounter  variables  in  the  Bayesian  network  A  C  X  from  a  distribution 
Pait(A)  that  is  different  from  the  encounter  model  distribution  P(X),  special  care  must  be  taken 
when  sampling  the  remaining  variables  so  that  the  relationships  in  the  Bayesian  networks  are 
preserved.  In  other  words,  after  sampling  the  variables  in  A  sampling  then  occurs  over  the  remaining 
variables  B  =  X  \  A  according  to  P{B  |  A)  as  inferred  from  the  Bayesian  network. 

Simply  assigning  variables  in  A  to  values  according  to  Halt  (A)  is  permissible  so  long  as  the  remaining 
variables  B  come  later  in  a  topological  sort  of  X. 

Another  way  to  ensure  that  the  distribution  of  the  remaining  variables  reflects  the  encounter  model 
distribution  is  to  use  logic  sampling  [16].  First,  sample  the  variables  in  A  according  to  Halt  (A). 
Then,  sample  from  the  encounter  model  Bayesian  network  until  a  sample  is  obtained  that  matches 
the  bins  that  were  assigned  from  the  alternative  distribution.  The  probabilistic  logic  sampling 
approach  can  require  the  generation  of  many  samples  before  acceptance. 

Alternately,  one  can  manipulate  the  graphical  structure  of  the  encounter  model  Bayesian  network 
to  make  the  variables  in  A  appear  earlier  than  the  variables  in  B  in  a  topological  sort.  Then,  the 
variables  in  A  can  be  assigned  directly  according  to  Hait(A)  and  then  the  remaining  variables  B 
can  be  assigned  using  forward  sampling.  Special  care  must  be  taken  when  changing  the  direction  of 
edges  (also  called  arcs)  in  the  Bayesian  network  to  preserve  the  relationships  between  variables.  If 
one  wishes  to  reverse  the  direction  of  an  edge  between  two  nodes  A;  and  Xj  in  a  Bayesian  network, 
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one  must  add  the  parents  of  Xj  to  Xi  and  vice  versa.  This  arc  reversal  operation  preserves  the 
original  distribution  over  X  [17, 18].  A  downside  to  arc  reversal  is  that  the  resulting  network  can 
become  very  complex. 


5.4.2  Likelihood  Weighting 

Other  analyses  may  require  that  encounters  being  generated  from  a  distribution  that  cannot  be 
accounted  for  by  modifying  the  Bayesian  network.  For  example,  an  analysis  may  require  that  the 
distribution  over  HMD  be  different  than  the  distribution  generated  by  the  model.  Note  that  HMD 
is  not  a  variable  in  the  Bayesian  network,  but  is  a  product  of  sampling  from  the  network  and 
initializing  the  encounter.  If  this  is  the  case,  then  each  sample  should  be  weighted  according  to 
the  likelihood  of  a  particular  HMD  value  occurring  in  the  airspace  divided  by  the  likelihood  of 
generating  that  value  of  HMD  using  the  model.  More  generally, 
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where  yW  represents  the  value  of  the  variable  (or  variables)  under  consideration  for  likelihood 
weighting,  g(y^)  is  the  density  over  y  that  occurs  in  the  airspace,  and  /( yW)  is  the  sampling 
density  of  y  generated  by  the  model. 
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6.  SUMMARY 


This  document  described  the  development  of  a  due  regard  encounter  model.  This  model  should 
be  used  to  evaluate  SAA  systems  operating  on  UAS  flying  due  regard  in  international  airspace. 
This  model  assumes  that  aircraft  blunder  into  close  proximity  without  prior  intervention.  In  this 
model  the  trajectories  of  the  aircraft  involved  in  the  encounter  are  independent  of  each  other  prior 
to  collision  avoidance  or  SAA  system  intervention. 

The  process  to  develop  this  model  closely  follows  the  approach  to  develop  other  recent  encounter 
models.  The  approach  involved  modeling  aircraft  dynamics  as  a  Markov  process  where  the  state  of 
the  aircraft  in  the  future  probabilistically  depends  on  its  current  state.  The  model  is  encoded  in  a 
dynamic  Bayesian  network,  which  efficiently  captures  the  statistical  dependencies  between  variables 
in  the  model. 

The  key  difference  between  this  model  and  others  is  the  data  used  to  build  the  model.  Previous 
models  have  used  radar  data  with  5  or  12  s  updates.  However,  this  model  is  based  on  ETMS  data 
with  a  60  s  update.  Part  of  this  model  development  involved  validating  that  ETMS  data  can  be 
used  to  build  a  due  regard  encounter  model. 
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APPENDIX  A 

VARIABLE  DISTRIBUTIONS 


Figures  A-l  through  A- 7  show  marginal  distributions  for  the  variables  in  the  due  regard  encounter 
model. 
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Figure  A-l.  Altitude  layer  distribution. 
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Figure  A-2.  Geographic  domain  distribution. 
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Figure  AS. 


Airspeed  distribution  (kts). 
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Figure  A-J^.  Heading  distribution  (°). 
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Figure  A-5.  Acceleration  distribution  (kts/s). 
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Figure  A- 6.  Turn  rate  distribution  (°/s). 
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APPENDIX  B 

VALIDATION  OF  ETMS  DATA 


Previous  encounter  models  have  been  built  using  raw  radar  data  that  estimate  the  position  of 
aircraft  once  every  5  or  12  s.  In  order  to  build  previous  encounter  models,  the  raw  radar  data  are 
processed  to  extract  tracks.  Outliers  are  removed  from  the  tracks,  then  the  tracks  are  smoothed 
and  interpolated.  Finally,  variables  in  the  encounter  model  are  estimated. 

The  due  regard  encounter  model,  that  is  the  subject  of  this  report,  is  built  using  ETMS  data.  There 
are  some  important  differences  between  ETMS  data  and  radar  data.  First,  ETMS  data  includes 
processed  observations  of  aircraft  that  are  reported  once  a  minute.  Second,  the  ETMS  data  includes 
estimates  of  aircraft  ground  speed  and  heading.  The  differences  between  radar  data  and  ETMS 
data  raise  questions  about  the  suitability  of  ETMS  data  for  building  an  encounter  model.  This 
Appendix  describes  a  validation  of  the  ETMS  data. 

Section  B.l  describes  the  approach  to  validate  that  the  ETMS  data  feed  can  be  used  to  build  an 
encounter  model.  Section  B.2  discusses  the  results. 


B.l  APPROACH 

The  validation  approach  is  shown  in  Figure  B-l.  First,  both  radar  and  ETMS  data  are  collected. 
Because  the  purpose  here  is  to  validate  ETMS  data  for  building  a  due  regard  encounter  model, 
tracks  were  selected  near  the  AN/FPS-117  BAR  radar  in  Barrington,  Nova  Scotia,  Canada.  It  is  a 
long  range  radar  with  a  maximum  range  of  approximately  250  NM  and  has  coverage  of  a  portion 
of  the  NAT.  This  radar  is  located  at  43°27'7//  N  and  65°28/20,/  W.  Once  tracks  within  250  NM 
of  BAR  are  selected  from  the  Radar  Evaluation  Squadron  (RADES)  and  ETMS  data  feeds,  the 
tracks  are  associated  with  each  other.  Any  track  that  is  found  in  both  data  sets  is  kept  and  all 
other  tracks  are  removed.  Next,  encounter  models  are  constructed  from  both  sets  of  data  and  the 
two  models  are  compared.  In  total,  the  two  models  were  each  built  with  1775  hours  of  flight  data. 


Figure  B-2  shows  a  5-minute  window  of  tracks  near  the  BAR  radar.  The  red  marks  indicate  tracks 
from  the  RADES  dataset,  while  blue  marks  indicate  tracks  from  the  ETMS  tracks. 


31 


Figure  B-l.  ETMS  validation  process. 
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Figure  B-2.  Five  minute  window  of  tracks. 
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B.2  RESULTS 


To  compare  the  two  encounter  models,  the  Bayes’  similarity  scores  was  calculated,  which  is  a 
measure  of  the  similarity  of  two  Bayesian  networks  [7].  If  the  similarity  score  is  positive,  then  the 
two  networks  are  likely  the  same.  However,  if  the  score  is  negative,  then  the  two  networks  are  likely 
different.  Table  B-l  shows  the  validation  results.  These  results  indicate  that  both  networks  are 
different.  Because  the  magnitude  for  the  score  of  the  transition  networks  is  lower,  the  transition 
networks  are  more  similar  to  each  other. 


TABLE  B-l 
Validation  results. 


Initial  Network  —186441 
Transition  Network  —564 


Figures  B-3  through  B-6  show  marginalized  distributions  for  airspeed,  airspeed  acceleration,  turn 
rate,  and  vertical  rate.  Distributions  for  the  RADES  encounter  model  are  in  red,  while  the  ETMS 
encounter  model  variables  are  blue.  Further  inspection  of  these  variables  for  both  networks  show 
that  distribution  for  these  variables  are  very  similar.  Even  though  the  networks  are  statistically 
different,  they  are  assumed  to  be  sufficiently  similar  for  evaluating  a  SAA  system.  Thus,  one  can 
conclude  that  the  ETMS  data  feed  is  acceptable  for  building  an  encounter  model. 
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Speed  (kt) 

Figure  B-3.  Airspeed  distribution  comparison. 
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Figure  B-J^.  Acceleration  distribution  comparison. 
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Figure  B-5.  Turn  rate  distribution  comparison. 
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Figure  B-6.  Vertical  rate  distribution  comparison. 
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APPENDIX  C 

MODEL  VALIDITY  LENGTH 


The  model  validity  length  (MVL)  is  defined  as  the  length  of  time  that  simulated  tracks  generated 
from  the  encounter  model  are  representative  of  the  observed  tracks;  it  is  one  method  of  deducing 
the  stochastic  stability  or  “validity”  of  the  model.  Validity  is  assessed  by  comparing  the  marginal 
feature  distributions  (e.g.,  airspeed,  acceleration,  and  turn  rate)  of  sampled  tracks  to  the  observed 
distributions.  Airspeed  is  important  to  analyze  because  it  is  not  explicitly  defined  in  the  model’s 
transition  network,  but  is  rather  propagated  in  a  dynamic  simulation  by  the  airspeed  acceleration. 
Because  the  airspeed  is  the  time  integral  of  the  acceleration,  it  is  unbounded  unless  limits  are 
enforced  in  the  dynamic  simulation  environment. 


C.l  APPROACH 


This  process  to  determine  the  MVL  starts  by  sampling  tracks  from  the  encounter  model.  The 
aircraft  trajectories  are  then  propagated  using  a  dynamic  model  and  feature  distributions  at  each 
time  step  are  captured.  Features  are  then  discretized  and  counted  into  bins,  with  bin  outpoints 
defined  as  in  the  model.  The  result  is  a  time  history  of  the  discrete  feature  distributions  for  the 
set  of  tracks.  These  distributions  are  compared  to  those  originally  observed  in  the  NAS.  The  first 
time  step  at  which  a  simulated  distribution  is  no  longer  representative  of  the  observed  distribution 
is  considered  the  MVL.  The  distributions  are  visually  and  statistically  compared. 

For  a  statistical  comparison,  the  general  goodness-of-fit  test  is 

X2  =  E  ~  "*■  e)2  ,  (C-1) 

k= i  nk’E 

where  K  is  the  number  of  bins,  is  the  bin  count,  and  the  subscripts  S  and  E  denote  simu¬ 
lated  and  expected  (observed),  respectively.  The  x2  distribution  has  K- 1  degrees  of  freedom.  As 
the  number  of  samples  increases  beyond  approximately  10,000,  %2  also  increases  so  that  minor 
deviations  between  two  distributions  may  become  statistically  significant  [19].  To  overcome  this, 
Hamada  et  al.  suggest  using  a  sample  size  (n)  proportional  to  K  (such  as  K  ~  n0A)  [20].  In 
contrast  to  using  a  small  sample  size,  the  sample  size  (Nm)  is  artificially  reduced  when  calculating 
X2  to  K2'5.  This  modifies  the  calculation  of  x2  to 


X 


2 


K 


NmJ2 

k=  1 


(Pk,S  -  Pk,E)2 
Pk,E 


(C-2) 


where  Nm  is  the  modified  sample  size  proportional  to  K,  and  p  =  n/N  where  N  is  the  original 
sample  size. 

The  p- value  resulting  from  the  test  is  the  probability  of  obtaining  the  sample’s  x2  test  statistic  or 
one  more  extreme  by  chance.  The  closer  the  p-value  is  to  zero,  the  smaller  the  probability  that  the 
simulated  distribution  is  similar  to  the  expected  distribution  observed  in  the  NAS. 
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Given  the  time  history  of  p-values,  the  MVL  is  defined  using  a  threshold  p-value;  the  time  at  which 
the  p-value  decreases  past  this  threshold  is  considered  the  MVL.  Though  the  choice  of  threshold 
is  somewhat  arbitrary,  p-values  of  0.05  or  0.01  are  typical  choices;  0.01  is  used  here.  The  p-value 
threshold  corresponds  to  the  maximum  allowable  Type-I  error — the  probability  that  the  results  are 
due  to  chance  alone.  It  is  important  to  note  that  p-values  resulting  from  statistical  testing  can  be 
misleading  and  must  be  interpreted  with  care  [19]. 

C.2  RESULTS 

Figure  C-l  shows  the  MVL  p-values  as  a  function  of  time.  At  1000  s  simulation  time,  the  p-value 
is  0.913.  This  result  indicates  that  the  distribution  of  airspeeds  simulated  by  the  model  are  very 
stable.  Thus,  the  encounter  model  can  accommodate  simulations  of  encounters  that  last  at  least 
1000  s.  Figure  C-2  shows  histograms  of  airspeed  at  200  s  intervals  generated  by  the  encounter 
model.  These  results  agree  with  the  findings  in  Figure  C-l. 
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Figure  C-l.  Model  validity  length  results. 
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Figure  C-2.  Simulated  airspeed  distributions. 
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